TrustRadius Insights
Apache Kafka is a widely-used platform that has proven to be invaluable in various industries and applications. It is relied upon by …
Overview
What is Apache Kafka?
Apache Kafka is an open-source stream processing platform developed by the Apache Software Foundation written in Scala and Java. The Kafka event streaming platform is used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical…
Recent Reviews
Reviewer Pros & Cons
Product Details
- About
- Tech Details
What is Apache Kafka?
Apache Kafka is an open-source stream processing platform developed by the Apache Software Foundation written in Scala and Java. The Kafka event streaming platform is used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.
Apache Kafka Technical Details
| Operating Systems | Unspecified |
|---|---|
| Mobile Application | No |
Comparisons
Compare with
Reviews and Ratings
(127)
Community Insights
TrustRadius Insights are summaries of user sentiment data from TrustRadius reviews and, when necessary, 3rd-party data sources. Have feedback on this content? Let us know!
- Business Problems Solved
- Pros
- Cons
- Recommendations
Apache Kafka is a widely-used platform that has proven to be invaluable in various industries and applications. It is relied upon by organizations to have real-time communication and keep order information up-to-date. This is particularly useful for organizations that need to process large volumes of data, such as those in the cybersecurity industry. Apache Kafka is also considered the go-to tool for event streaming, generating events and notifying relevant applications for consumption. Additionally, it is used in both first-party and third-party components of applications to address data proliferation and enable efficient notifications.
Another key use case for Apache Kafka is replacing classical messaging software within organizations, becoming the new standard for messaging. This powerful streaming framework plays a crucial role as a queuing mechanism for records in various pipelines, providing a simple yet efficient system for queuing and maintaining records. Moreover, Apache Kafka excels at storing and processing records in dedicated servers, supporting high data loads and offering the ability to replay consumed data. This makes it ideal for buffering incoming records during traffic spikes or in case of data infrastructure failures.
Furthermore, Apache Kafka finds its purpose in driving real-time monitoring by sending log information to feed other applications. Its ability to scale and manage common errors in messaging allows organizations to handle large quantities of messages per second without compromising performance. Another notable use case involves Apache Kafka acting as an efficient stream/message ingestion engine for customer-facing applications, enabling internal analytics and real-time decision-making.
Additionally, Apache Kafka integrates seamlessly with big data technologies like Spark, making it a valuable addition to big data ecosystems. Organizations have successfully replaced legacy messaging solutions with Apache Kafka, thanks to its ability to serve as a messaging and data-streaming pipeline solution. It enables modern streaming API-based applications while ensuring high availability and clustering as a message broker between client-facing applications.
Moreover, Apache Kafka serves as an ingress and egress queue for big data systems, facilitating data storage and retrieval processes. It also acts as a reliable queue for frontend applications to retrieve data and analytics from MapR and HortonWorks. With over five years of being utilized in data pipelines, Apache Kafka has consistently demonstrated excellent performance and reliability.
In summary, Apache Kafka proves to be versatile and essential across various industries and use cases. It facilitates real-time communication, ensures data integrity, enables efficient event streaming, replaces classical messaging software, and supports high scalability and fault tolerance. With its robust capabilities, Apache Kafka continues to be the go-to solution for organizations seeking to streamline their data processing and communication systems.
Fault tolerance and high scalability: Users have consistently praised Apache Kafka for its fault tolerance and high scalability. Many reviewers have stated that Kafka excels in handling large volumes of data and is considered a workhorse in data streaming.
Ease of administration: Reviewers appreciate Kafka's ease of administration, noting that it offers an abundance of options for managing and maintaining queues. Multiple users have mentioned that the platform allows for easy expansion and configuration of cluster growth, making it straightforward to administer.
Real-time streaming capabilities: Kafka's real-time streaming capabilities are seen as a significant advantage by users. Several reviewers have highlighted the platform's ability to handle real-time data pipelines and its resistance to node failure within the cluster. This feature enables users to process asynchronous data efficiently and ensures continuous availability of the system.
Difficulty Monitoring Kafka Deployments: Some users have found it difficult to monitor their Kafka deployments and have expressed a desire for a separate monitoring dashboard that would provide them with better visibility into their topics and messages.
Steep Learning Curve for Creating Brokers and Topics: The process of creating brokers and topics in Kafka has been described as having a steep learning curve by some users, who believe that it could be simplified to make it more accessible.
Outdated Web User Interface: The web user interface of Kafka has not been updated in years, leading some users to feel that it lacks a streamlined user experience. They express the need for a more modern interface instead of relying on third-party tools.
Users have recommended using Apache Kafka for various messaging platform requirements. It integrates easily with multiple programming languages, offers stream processing capabilities, distributed data storage, and the ability to handle multiple requests simultaneously.
Another common recommendation is to consider Apache Kafka as a messaging broker due to its extensive feature set and guaranteed delivery of data to consumers. Users find it highly supported and widely used within the community.
Users also recommend Apache Kafka for streaming large amounts of data. They praise its scalability and ease of use, although they mention that manual rebalancing of partitions may be required when adding or deleting nodes. Additionally, users appreciate that Kafka allows connections between multiple producers and consumers with low resource consumption.
Overall, Apache Kafka is regarded as a practical choice for message processing systems, data streaming, and handling large volumes of data due to its stability, scalability, and diverse features.
Attribute Ratings
Reviews
(1-5 of 5)Companies can't remove reviews or game the system. Here's why
August 23, 2023
Apache Kafka - Default Choice For Large Scale Messaging
Score 8 out of 10
Vetted Review
Verified User
Apache Kafka is really the bedrock of all things streaming and data processing. I cannot imagine if there is any other product that does it better. My last 2 companies used it, and my current one does so as well. If you want your data stream to be organized and sent, Apache Kafka has become the tool of choice. I have dabbled in Azure EventHubs as well, if you are into opensource data streaming, Apache Kafka will take you where you need to be for data lakes and the amount of data that is streamed for the cybersecurity industry that my company is in. Without Apache Kafka, there is no way that my company products can handle the volume of data that we process for our customers.
- Data streaming is really second to none.
- Scaling, done right, Apache Kafka is a workhorse.
- Ease of administration - Although you cannot really compare to Azure EventHubs, but that is comparing between Apples and Oranges.
- The web UI has not really changed in years. UX has been refreshed, but a more streamlined UX instead of many 3rd party webUX tools, will be most welcome.
- Webhooks can still be tricky to troubleshoot at times.
- CLI monitoring is a learning curve to get it right.
- Well known and known set of tools from setup to admin.
- Scalability.
- Fit for use in both onprem, and cloud-base use cases.
- Being an open-source tool, Apache Kafka is invaluable to my company's product. I cannot imagine how much it is if we are using Amazon Kinesis or Azure EventHubs.
- The negative part will be in the event of Apache Kafka failures, the trouble-shooting can really be a pain and bane. But given enough exposure to its inner workings, Apache Kafka still comes out OK.
- Having used Apache Kafka for years in this company, I can only say without Apache Kafka, my company would not be cost-efficient and would be much more costlier to sell to customers if we were paying on top of Azure Event hubs or Amazon Kinesis.
Apache Kafka is built for scale. From high throughput and real-time data streaming, it has a strong advantage over RabbitMQ with its low latency. This put Apache Kafka at the forefront as the platform of choice for large datasets messaging and ensuring scalability when data scale up tremendously.
RabbitMQ however has its strengths in traditional messaging. Routing and message delivery reliability are the bedrock of RabbitMQ and this is where RabbitMQ excels. In my previous workplace, RabbitMQ was of choice as reliability matters more than scale.
In two words. Apache Kafka for scale, RabbitMQ for reliability. And for cloud deployment and large dataset messaging in what I am doing now, Apache Kafka is the default choice.
RabbitMQ however has its strengths in traditional messaging. Routing and message delivery reliability are the bedrock of RabbitMQ and this is where RabbitMQ excels. In my previous workplace, RabbitMQ was of choice as reliability matters more than scale.
In two words. Apache Kafka for scale, RabbitMQ for reliability. And for cloud deployment and large dataset messaging in what I am doing now, Apache Kafka is the default choice.
May 30, 2023
Kafka for tracking changes
Score 8 out of 10
Vetted Review
Verified User
We use Apache Kafka to stream order information across systems. An order may go through certain updates through its lifecycle. These updates need to be communicated to the systems in near real time and we rely on Kafka for this.Our business use case is to take these orders up with the insurance companies for approval and thus the order information need to be up to date. Kafka has been excellent at doing this so far.
- Receiving messages from publisher and sending to consumer in FIFO manner
- Handling of errors using Dead Letter Queue when message could not be consumed on the consumer end
- Fault tolerance
- Sometimes it becomes difficult to monitor our Kafka deployments. We've been able to overcome it largely using AWS MSK, a managed service for Apache Kafka, but a separate monitoring dashboard would have been great.
- Simplify the process for local deployment of Kafka and provide a user interface to get visibility into the different topics and the messages being processed.
- Learning curve around creation of broker and topics could be simplified
- High throughput
- Low latency
- Fault tolerance
- We are able to submit orders to the insurance companies with almost 100% accuracy because we receive Kafka updates in almost real time
- We are getting notified of error scenarios separately because of our Dead Letter Queue implementation so that we can handle those cases
- There is certain engineering effort being spent to maintain Kafka
Apache Kafka can work at a higher scale as compared to SQS. It can work with higher size per message and millions of messages per second. Moreover it can be scaled horizontally by adding more brokers to the cluster. SQS is good enough for simple use cases like making a task async by passing it to the worker app or delaying a task execution by certain time duration but not advisable for heavy load systems.
1000
We have an engineering team of about 1500 engineers and almost 2/3 of them use Kafka for some or other use case.
50
Support for Kafka comes from within our developer community. So the people managing Kafka are engineers only.
- Publishing messages to other services to inform them of modification in an entity
- Publishing messages to Camunda to mark a user task as complete
August 07, 2021
Apache Kafka for your Data solutions
It is being used for the product mainly. We have huge data pipelines running which depend on Apache Kafka. It is being used for more than 5 years now and we are really happy with the performance and the reliability Apache Kafka has to offer. The experience has been excellent.
- Data Pipeline
- Asynchronous processing
- Data retention for reprocessing
- Dashboards to monitor the performance
- ZooKeeper free
- Connectors for more languages
- Faster deployments
- Scalable solution which improves up time of systems
- Low Dev effort
Apache Kafka is much more scalable and more reliable. Does not depend on memory, works well on rotational disks and that makes it a cheaper to use solution on low hardware requirements. Running multiple consumers on the same topic can also mean processing the same data again and again and this can be a big plus.
April 09, 2021
Apache Kafka: Where messaging meets storage
Apache Kafka is used by our company as the "next generation" of messaging/data-streaming pipeline solutions, to replace our old legacy JMS-based messaging solution and enable the modern streaming API based applications. When it is used for messaging purposes, we shift the responsibility of data replay from the message source (publisher application) to the message destination (consumer application). This flexibility resolved the legacy issue of sources replaying the messages but impacting all subscribers to the same topic. When Kafka is used as the streaming pipeline, it is integrated seamlessly with the Spark/Spring Stream-based analytic solutions, as it is also a kind of distributed storage.
- Undoubtedly, Kafka's high throughput and low latency feature are the highlights.
- Kafka can scale horizontally very well.
- The CLI and configuration details need to be worked out more in-depth. The naming convention of configuration is not so good and causing a lot of confusion. Sometimes there are too many configuration parameters to tune--requires the adopter to understand a lot of tricks like NFS entrapment, for example.
- Lack of a good monitoring solution so far
- Kafka makes the messaging itself more reliable (as it has the distributed storage by itself and the message doesn't disappear even after it has been consumed).
- Kafka can support a much higher volume use case, without too much extra pressure on the existed hardware.
Kafka is not a real messaging broker implementation as RabbitMQ or TIBCO EMS/JMS are. Although it can be used as messaging, we like the idea behind the Kafka (data isn't "passing by," instead it remains centra, so the client can revisit the data if necessary). This also relieves the pressure of keeping the old duplicated data copy on both the publisher and the consumer sides.
March 01, 2018
Apache Kafka, the F1 of messaging
Apache Kafka is becoming the new standard for messaging at our organization. Originally we limited the use to big data environments and projects but as the technology is becoming more mature we think it will eventually replace classical messaging software.
- High volume/performance throughput environments
- Low latency projects
- Multiple consumers for the same data, reprocessing, long-lasting information
- Still a bit inmature, some clients have required recoding in the last few versions
- New feaures coming very fast, several upgrades a year may be required
- Not many commercial companies provide support
- Easier deployment and horizontal scalability
- Messaging cost reduction
- Developments require adaptation and some paradigm shift to interoperate with Kafka
- IBM MQ and TIBCO EMS
Kafka is faster and more scalable, also "free" as opensource (albeit we deploy using a commercial distribution). Infrastructure tends to be cheaper. On the other hand, projects must adapt to Kafka APIs that sometimes change and BAU increases until a major 1.x version comes out and adds stability to the product.
20
Kafka is core for several business/technical functions:
- Data streaming: ingest data into Datalake, process information near real-time
- Log processing: able to hold logs from all the company applications both to process and as transport them to a final storage (like timeseries DB, Elasticsearch and so on)
- Reliable messaging now that exactly-one-delivery semantics have been implemented
and so on
- Data streaming: ingest data into Datalake, process information near real-time
- Log processing: able to hold logs from all the company applications both to process and as transport them to a final storage (like timeseries DB, Elasticsearch and so on)
- Reliable messaging now that exactly-one-delivery semantics have been implemented
and so on
Developers with deep knowledge of stream processing, otherwise your organization will not use all the capabilities.
Operators with DevOps skills. Kafka, even in version 1.0, is still a bit inmature and lacks of proper adminitration tools (apart from those from 3rd party like Cloudera, Hortonworks, Confluent, Lenses and so on) so hands on scripts and detailled monitoring of the platform is a must.
Operators with DevOps skills. Kafka, even in version 1.0, is still a bit inmature and lacks of proper adminitration tools (apart from those from 3rd party like Cloudera, Hortonworks, Confluent, Lenses and so on) so hands on scripts and detailled monitoring of the platform is a must.
- Application technical log processing
- Realtime transaction analysis
- Messaging as a Service for PaaS and CaaS applications
- Several data hubs: technical, business, social...